Augmented reality environments allow interaction among users and both real-world objects and virtual or digital objects. To achieve these augmented reality environments, a system may project content onto projection surfaces, such walls or a portable projection surface, within an environment. In addition, this system may monitor the environment for user gestures, in response to which the system may perform predefined actions. As such, such a system may monitor the environment for certain objects of interest, such as a projection surface (for projecting content thereon) or a user's hand (for the purpose of identifying a user's gesture). However, because tracking objects in this manner may be computationally expensive, the system may have difficulty tracking these objects in real-time or near-real-time, thus lessening the experience of a user employing the system.